20/30

# 12 Replacing a Server Disk

The failure of a disk is never catastrophic on Oracle Big Data Appliance. No user data is lost. Data stored in HDFS or Oracle NoSQL Database is automatically replicated.

This chapter contains the following sections:

- Overview of the Disk Replacement Process

- About Disk Drive Identifiers

- Prerequisites for Replacing a Failing Disk

- What If a Server Fails to Restart?

- Replacing a Disk Drive

- Correcting a Mounted Partitions Error

- Identifying the Function of a Disk Drive

- Configuring an Operating System Disk

- Configuring an HDFS or Oracle NoSQL Database Disk

- Verifying the Disk Configuration

See Also:

My Oracle Support Doc ID 1581331.1

## 12.1 Overview of the Disk Replacement Process

The following are the basic steps for replacing a server disk drive:

1. Replace the failed disk drive.

2. Perform the basic configuration steps for the new disk.

3. Identify the dedicated function of the failed disk, as an operating system disk, or as either an HDFS disk or an Oracle NoSQL Database disk.

4. Configure the disk for its dedicated function.

5. Verify that the configuration is correct.

6. Install the Oracle Big Data Appliance software.

See Also:

"Servicing Storage Drives and Rear Drives" in the Sun Server X4-2L Service Manual at

`https://docs.oracle.com/cd/E36974_01/html/E38145/z40000091011460.html#scrolltoc`

"Servicing Storage Drives and Boot Drives" in the Sun Fire X4270M2 Server Service Manual at

`http://docs.oracle.com/cd/E19245-01/E21671/hotswap.html#50503714_61628`

## 12.2 About Disk Drive Identifiers

The Oracle Big Data Appliance servers contain a disk enclosure cage that is controlled by the host bus adapter (HBA). The enclosure holds 12 disk drives that are identified by slot numbers 0 to 11. The drives can be dedicated to specific functions, as shown in Table 12-1.

Oracle Big Data Appliance uses symbolic links, which are defined in `/dev/disk/by_hba_slot`, to identify the slot number of a disk. The links have the form snpm, where n is the slot number and m is the partition number. For example, `/dev/disk/by_hba_slot/s0p1` initially corresponds to `/dev/sda1`.

When a disk is hot swapped, the operating system cannot reuse the kernel device name. Instead, it allocates a new device name. For example, if you hot swap `/dev/sda`, then the disk corresponding `/dev/disk/by-hba-slot/s0` might link to `/dev/sdn` instead of `/dev/sda`. Therefore, the links in `/dev/disk/by-hba-slot/` are automatically updated when devices are added or removed.

The command output lists device names as kernel device names instead of symbolic link names. Thus, `/dev/disk/by-hba-slot/s0` might be identified as `/dev/sda` in the output of a command.

### 12.2.1 Standard Disk Drive Mappings

Table 12-1 shows typical initial mappings between the RAID logical drives and the operating system identifiers. Nonetheless, you must use the mappings that exist for your system, which might be different from the ones listed here. The table also identifies the dedicated function of each drive in an Oracle Big Data Appliance server. The server with the failed drive is part of either a CDH cluster (HDFS) or an Oracle NoSQL Database cluster.

Table 12-1 Disk Drive Identifiers

Symbolic Link to Physical Slot | Typical Initial Kernel Device Name | Dedicated Function
--- | --- | ---
/dev/disk/by-hba-slot/s0 | /dev/sda | Operating system
/dev/disk/by-hba-slot/s1 | /dev/sdb | Operating system
/dev/disk/by-hba-slot/s2 | /dev/sdc | HDFS or Oracle NoSQL Database
/dev/disk/by-hba-slot/s3 | /dev/sdd | HDFS or Oracle NoSQL Database
/dev/disk/by-hba-slot/s4 | /dev/sde | HDFS or Oracle NoSQL Database
/dev/disk/by-hba-slot/s5 | /dev/sdf | HDFS or Oracle NoSQL Database
/dev/disk/by-hba-slot/s6 | /dev/sdg | HDFS or Oracle NoSQL Database
/dev/disk/by-hba-slot/s7 | /dev/sdh | HDFS or Oracle NoSQL Database
/dev/disk/by-hba-slot/s8 | /dev/sdi | HDFS or Oracle NoSQL Database
/dev/disk/by-hba-slot/s9 | /dev/sdj | HDFS or Oracle NoSQL Database
/dev/disk/by-hba-slot/s10 | /dev/sdk | HDFS or Oracle NoSQL Database
/dev/disk/by-hba-slot/s11 | /dev/sdl | HDFS or Oracle NoSQL Database

### 12.2.2 Standard Mount Points

Table 12-2 show the mappings between HDFS partitions and mount points.

Table 12-2 Mount Points

Symbolic Link to Physical Slot and Partition | HDFS Partition | Mount Point
--- | --- | ---
/dev/disk/by-hba-slot/s0p4 | /dev/sda4 | /u01
/dev/disk/by-hba-slot/s1p4 | /dev/sdb4 | /u02
/dev/disk/by-hba-slot/s2p1 | /dev/sdc1 | /u03
/dev/disk/by-hba-slot/s3p1 | /dev/sdd1 | /u04
/dev/disk/by-hba-slot/s4p1 | /dev/sde1 | /u05
/dev/disk/by-hba-slot/s5p1 | /dev/sdf1 | /u06
/dev/disk/by-hba-slot/s6p1 | /dev/sdg1 | /u07
/dev/disk/by-hba-slot/s7p1 | /dev/sdh1 | /u08
/dev/disk/by-hba-slot/s8p1 | /dev/sdi1 | /u09
/dev/disk/by-hba-slot/s9p1 | /dev/sdj1 | /u10
/dev/disk/by-hba-slot/s10p1 | /dev/sdk1 | /u11
/dev/disk/by-hba-slot/s11p1 | /dev/sdl1 | /u12

### 12.2.3 Obtaining the Physical Slot Number of a Disk Drive

Use the following `MegaCli64` command to verify the mapping of virtual drive numbers to physical slot numbers. See "Replacing a Disk Drive."

    # MegaCli64 LdPdInfo a0 | more

## 12.3 Prerequisites for Replacing a Failing Disk

To replace an HDFS disk or an operating system disk that is in a state of predictive failure, you must first dismount the HDFS partitions. You must also turn off swapping before replacing an operating system disk.

Note:

Only dismount HDFS partitions. For an operating system disk, ensure that you do not dismount operating system partitions. Only partition 4 (sda4 or sdb4) of an operating system disk is used for HDFS.

To dismount HDFS partitions:

1. Log in to the server with the failing drive.

2. If the failing drive supported the operating system, then turn off swapping:

       # bdaswapoff

   Removing a disk with active swapping crashes the kernel.

3. List the mounted HDFS partitions:

       # mount -l

       /dev/md2 on / type ext4 (rw,noatime)
       proc on /proc type proc (rw)
       sysfs on /sys type sysfs (rw)
       devpts on /dev/pts type devpts (rw,gid=5,mode=620)
       /dev/md0 on /boot type ext4 (rw)
       tmpfs on /dev/shm type tmpfs (rw)
       /dev/sda4 on /u01 type ext4 (rw,nodev,noatime) [/u01]
       /dev/sdb4 on /u02 type ext4 (rw,nodev,noatime) [/u02]
       /dev/sdc1 on /u03 type ext4 (rw,nodev,noatime) [/u03]
       /dev/sdd1 on /u04 type ext4 (rw,nodev,noatime) [/u04]
            .
            .
            .

4. Check the list of mounted partitions for the failing disk. If the disk has no partitions listed, then proceed to "Replacing a Disk Drive." Otherwise, continue to the next step.

   Caution:

   For operating system disks, look for partition 4 (sda4 or sdb4). Do not dismount an operating system partition.

5. Dismount the HDFS mount points for the failed disk:

       # umount mountpoint

   For example, `umount /u11` removes the mount point for partition `/dev/sdk1`.

   If the `umount` commands succeed, then proceed to "Replacing a Disk Drive." If a `umount` command fails with a device busy message, then the partition is still in use. Continue to the next step.

6. Open a browser window to Cloudera Manager. For example:

   `http://bda1node03.example.com:7180`

7. Complete these steps in Cloudera Manager:

   1. Log in as `admin`.

   2. On the Services page, click hdfs

   3. Click the Instances subtab.

   4. In the Host column, locate the server with the failing disk. Then click the service in the Name column, such as datanode, to open its page.

   5. Click the Configuration subtab.

   6. Remove the mount point from the Directory field.

   7. Click Save Changes.

   8. From the Actions list, choose Restart this DataNode.

   Note:

   If you removed the mount point in Cloudera Manager, then you must restore the mount point in Cloudera Manager after finishing all other configuration procedures.

8. Return to your session on the server with the failed drive.

9. Reissue the `umount` command:

       # umount mountpoint

10. Bring the disk offline:

        # MegaCli64 PDoffline "physdrv[enclosure:slot]" a0

    For example, `"physdrv[20:10]"` identifies disk s11, which is located in slot 10 of enclosure 20.

11. Delete the disk from the controller configuration table:

        MegaCli64 CfgLDDel Lslot a0

    For example, L10 identifies slot 10.

12. Complete the steps in "Replacing a Disk Drive."

## 12.4 What If a Server Fails to Restart?

The server may restart during the disk replacement procedures, either because you issued a `reboot` command or made an error in a MegaCli64 command. In most cases, the server restarts successfully, and you can continue working. However, in other cases, an error occurs so that you cannot reconnect using ssh. In this case, you must complete the reboot using Oracle ILOM.

To restart a server using Oracle ILOM:

1. Use your browser to open a connection to the server using Oracle ILOM. For example:

   `http://bda1node12-c.example.com`

   Note:

   Your browser must have a JDK plug-in installed. If you do not see the Java coffee cup on the log-in page, then you must install the plug-in before continuing.

2. Log in using your Oracle ILOM credentials.

3. Select the Remote Control tab.

4. Click the Launch Remote Console button.

5. Enter Ctrl+d to continue rebooting.

6. If the reboot fails, then enter the server `root` password at the prompt and attempt to fix the problem.

7. After the server restarts successfully, open the Redirection menu and choose Quit to close the console window.

See Also:

Oracle Integrated Lights Out Manager (ILOM) 3.0 documentation at

`http://docs.oracle.com/cd/E19860-01/`

## 12.5 Replacing a Disk Drive

Complete this procedure to replace a failed or failing disk drives.

1. Before replacing a failing disk, see "Prerequisites for Replacing a Failing Disk."

2. Replace the failed disk drive.

   See "Parts for Oracle Big Data Appliance Servers".

3. Power on the server if you powered it off to replace the failed disk.

4. Connect to the server as `root` using either the KVM or an SSL connection to a laptop.

5. Store the physical drive information in a file:

       # MegaCli64 pdlist a0 > pdinfo.tmp

   Note: This command redirects the output to a file so that you can perform several searches using a text editor. If you prefer, you can pipe the output through the `more` or `grep` commands.

   The utility returns the following information for each slot. This example shows a Firmware State of `Unconfigured(good), Spun Up`.

       Enclosure Device ID: 20
       Slot Number: 8
       Drive's postion: DiskGroup: 8, Span: 0, Arm: 0
       Enclosure position: 0
       Device Id: 11
       WWN: 5000C5003487075C
       Sequence Number: 2
       Media Error Count: 0
       Other Error Count: 0
       Predictive Failure Count: 0
       Last Predictive Failure Event Seq Number: 0
       PD Type: SAS
       Raw Size: 1.819 TB [0xe8e088b0 Sectors]
       Non Coerced Size: 1.818 TB [0xe8d088b0 Sectors]
       Coerced Size: 1.817 TB [0xe8b6d000 Sectors]
       Firmware state: Unconfigured(good), Spun Up
       Is Commissioned Spare : NO
       Device Firmware Level: 061A
       Shield Counter: 0
       Successful diagnostics completion on :  N/A
       SAS Address(0): 0x5000c5003487075d
       SAS Address(1): 0x0
       Connected Port Number: 0(path0)
       Inquiry Data: SEAGATE ST32000SSSUN2.0T061A1126L6M3WX
       FDE Enable: Disable
       Secured: Unsecured
       Locked: Unlocked
       Needs EKM Attention: No
       Foreign State: None
       Device Speed: 6.0Gb/s
       Link Speed: 6.0Gb/s
       Media Type: Hard Disk Device
       .
       .
       .

6. Open the file you created in Step 5 in a text editor and search for the following:

   - Disks that have a Foreign State of Foreign

   - Disks that have a Firmware State of Unconfigured

7. For disks that have a Foreign State of Foreign, clear that status:

       # MegaCli64 CfgForeign clear a0

   A foreign disk is one that the controller saw previously, such as a reinserted disk.

8. For disks that have a Firmware State of Unconfigured (Bad), complete these steps:

   1. Note the enclosure device ID number and the slot number.

   2. Enter a command in this format:

          # MegaCli64 pdmakegood physdrv[enclosure:slot] a0

      For example, [20:10] repairs the disk identified by enclosure 20 in slot 10.

   3. Check the current status of Foreign State again:

          # MegaCli64 pdlist a0 | grep foreign

   4. If the Foreign State is still Foreign, then repeat the `clear` command:

          # MegaCli64 CfgForeign clear a0

9. For disks that have a Firmware State of Unconfigured (Good), use the following command. If multiple disks are unconfigured, then configure them in order from the lowest to the highest slot number:

       # MegaCli64 CfgLdAdd r0[enclosure:slot] a0

       Adapter 0: Created VD 1

       Adapter 0: Configured the Adapter!!

       Exit Code: 0x00

   For example, [20:5] repairs the disk identified by enclosure 20 in slot 5.

10. If the `CfgLdAdd` command in Step 9 fails because of cached data, then clear the cache:

        # MegaCli64 discardpreservedcache l1 a0

11. Verify that the disk is recognized by the operating system:

        # lsscsi

    The disk may appear with its original device name (such as `/dev/sdc`) or under a new device name (such as `/dev/sdn`). If the operating system does not recognize the disk, then the disk is missing from the list generated by the `lsscsi` command.

    The `lssci` output might not show the correct order, but you can continue with the configuration. While the same physical to logical disk mapping is required, the same disk to device mapping for the kernel is not required. The disk configuration is based on `/dev/disks/by-hba-slot` device names.

    This example output shows two disks with new device names: `/dev/sdn` in slot 5, and `/dev/sdo` in slot 10.

        [0:0:20:0]   enclosu ORACLE  CONCORD14        0960  -
        [0:2:0:0]    disk    LSI      MR9261-8i        2.12  /dev/sda
        [0:2:1:0]    disk    LSI      MR9261-8i        2.12  /dev/sdb
        [0:2:2:0]    disk    LSI      MR9261-8i        2.12  /dev/sdc
        [0:2:3:0]    disk    LSI      MR9261-8i        2.12  /dev/sdd
        [0:2:4:0]    disk    LSI      MR9261-8i        2.12  /dev/sde
        [0:2:5:0]    disk    LSI      MR9261-8i        2.12  /dev/sdn
        [0:2:6:0]    disk    LSI      MR9261-8i        2.12  /dev/sdg
        [0:2:7:0]    disk    LSI      MR9261-8i        2.12  /dev/sdh
        [0:2:8:0]    disk    LSI      MR9261-8i        2.12  /dev/sdi
        [0:2:9:0]    disk    LSI      MR9261-8i        2.12  /dev/sdj
        [0:2:10:0]   disk    LSI      MR9261-8i        2.12  /dev/sdo
        [0:2:11:0]   disk    LSI      MR9261-8i        2.12  /dev/sdl
        [7:0:0:0]    disk    ORACLE   UNIGEN-UFD       PMAP  /dev/sdm
        [

12. Check the hardware profile of the server, and correct any errors:

        # bdacheckhw

13. Check the software profile of the server, and correct any errors:

        # bdachecksw

    If you see a "Wrong mounted partitions" error and the device is missing from the list, then you can ignore the error and continue. However, if you see a "Duplicate mount points" error or the slot numbers are switched, then see "Correcting a Mounted Partitions Error.".

14. Identify the function of the drive, so you configure it properly. See "Identifying the Function of a Disk Drive."

## 12.6 Correcting a Mounted Partitions Error

When the `bdachecksw` utility finds a problem, it typically concerns the mounted partitions.

An old mount point might appear in the `mount` command output, so that the same mount point, such as /u03, appears twice.

To fix duplicate mount points:

1. Dismount both mount points by using the `umount` command twice. This example dismounts two instances of/u03:

       # umount /u03
       # umount /u03

2. Remount the mount point. This example remounts /u03:

       # mount /u03

If a disk is in the wrong slot (that is, the virtual drive number), then you can switch two drives.

To switch slots:

1. Remove the mappings for both drives. This example removes the drives from slots 4 and 10:

       # MegaCli64 cfgLdDel L4 a0
       # MegaCli64 cfgLdDel L10 a0

2. Add the drives in the order you want them to appear; the first command obtains the first available slot number:

       # MegaCli64 cfgLdAdd [20:4] a0
       # MegaCli64 cfgLdAdd [20:5] a0

3. If mount errors persist even when the slot numbers are correct, then you can restart the server.

## 12.7 Identifying the Function of a Disk Drive

The server with the failed disk is configured to support either HDFS or Oracle NoSQL Database, and most disks are dedicated to that purpose. However, two disks are dedicated to the operating system. Before configuring the new disk, find out how the failed disk was configured.

Oracle Big Data Appliance is configured with the operating system on the first two disks.

To confirm that a failed disk supported the operating system:

1. Check whether the replacement disk corresponds to `/dev/sda` or `/dev/sdb`, which are the operating system disks.

       # lsscsi

   See the output from Step 11 of "Replacing a Disk Drive".

2. Verify that `/dev/sda` and `/dev/sdb` are the operating system mirrored partitioned disks:

       # mdadm -Q –-detail /dev/md2
       /dev/md2:
               Version : 0.90
         Creation Time : Mon Jul 22 22:56:19 2013
            Raid Level : raid1
            .
            .
            .
           Number   Major   Minor   RaidDevice State
              0       8        2        0      active sync   /dev/sda2
              1       8       18        1      active sync   /dev/sdb2

3. If the previous steps indicate that the failed disk is an operating system disk, then proceed to "Configuring an Operating System Disk."

## 12.8 Configuring an Operating System Disk

The first two disks support the Linux operating system. These disks store a copy of the mirrored operating system, a swap partition, a mirrored boot partition, and an HDFS data partition.

To configure an operating system disk, you must copy the partition table from the surviving disk, create an HDFS partition (ext4 file system), and add the software raid partitions and boot partitions for the operating system.

Complete these procedures after replacing the disk in either slot 0 or slot 1.

- Partitioning the Operating System Disk

- Repairing the RAID Arrays

- Formatting the HDFS Partition of an Operating System Disk

- Restoring the Swap Partition

- Restoring the GRUB Master Boot Records and HBA Boot Order

### 12.8.1 Partitioning the Operating System Disk

The partitioning procedure differs slightly between Oracle Linux versions 5 and 6. Follow the appropriate procedure for your system:

- Partitioning for Oracle Linux 6

- Partitioning for Oracle Linux 5

Note:

Replace `/dev/disk/by-hba-slot/s` `n` in the following commands with the appropriate symbolic link, either `/dev/disk/by-hba-slot/s0` or `/dev/disk/by-hba-slot/s1`.

#### 12.8.1.1 Partitioning for Oracle Linux 6

To partition a logical drive for Oracle Linux 6:

1. Complete the steps in "Replacing a Disk Drive."

2. Confirm that the new disk does not have a partition table:

       # parted /dev/disk/by-hba-slot/sn -s print

   You should see a message about a missing partition table.

3. If the `parted` command displays a partition table, then clear it:

       # dd if=/dev/zero of=/dev/disk/by-hba-slot/sn bs=1M count=100

   Tip:

   You can use this command to restart an operating system disk configuration, if you make a mistake.

4. Create the partition table:

       # parted /dev/disk/by-hba-slot/sn -s mklabel gpt print

5. List the Cylinder, Head, Sector (CHS) partition information of the surviving disk. Thus, if you are partitioning `/dev/disk/by-hba-slot/s0`, then enter `/dev/disk/by-hba-slot/s1` for `/dev/disk/by-hba-slot/s` `m` in the following command:

       # parted /dev/disk/by-hba-slot/sm -s unit chs print

       Model: LSI MR9261-8i (scsi)
       Disk /dev/sda: 243031,30,6
       Sector size (logical/physical): 512B/512B
       BIOS cylinder,head,sector geometry: 243031,255,63.  Each cylinder is 8225kB.
       Partition Table: gpt

       Number  Start         End           File system    Name     Flags
        1      0,0,34        25,127,7      ext4           primary  boot
        2      25,127,8      21697,116,20                 primary  raid
        3      21697,116,21  23227,61,35   linux-swap(v1) primary
        4      23227,61,36   243031,29,36  ext4           primary

6. Create partitions 1 to 3 on the new drive by duplicating the partitions of the surviving disk. Issue three commands in this format:

       # parted /dev/disk/by-hba-slot/sn -s mkpart primary file_system start end

   Use the start and end addresses that you obtained in Step 5 instead of the addresses shown in the following example:

       # parted /dev/disk/by-hba-slot/sn -s mkpart primary ext4 0,0,34 25,127,7
       # parted /dev/disk/by-hba-slot/sn -s mkpart primary ext4 25,127,8 21697,116,20
       # parted /dev/disk/by-hba-slot/sn -s mkpart primary linux-swap 21697,116,21 23227,61,35

7. Create primary partition 4 using the start address obtained in Step 5 and an end address of 100%:

       # parted /dev/disk/by-hba-slot/sn -s mkpart primary ext4 23227,61,36 100%

   Partition 4 stores HDFS data, and this syntax makes the partition as large as possible.

8. Set the boot flag:

       # parted -s /dev/disk/by-hba-slot/sn set 1 boot

9. Set the RAID flags:

       # parted -s /dev/disk/by-hba-slot/sn set 1 raid
       # parted -s /dev/disk/by-hba-slot/sn set 2 raid

10. Clear the names, using `parted` in interactive mode:

        # parted /dev/disk/by-hba-slot/sn

        GNU Parted 1.8.1
        Using /dev/sdn
        Welcome to GNU Parted! Type 'help' to view a list of commands.
        (parted) name 1 " "
        (parted) name 2 " "
        (parted) name 3 " "
        (parted) quit

11. Complete the steps in "Repairing the RAID Arrays".

#### 12.8.1.2 Partitioning for Oracle Linux 5

To partition a logical drive for Oracle Linux 5:

1. Complete the steps in "Replacing a Disk Drive."

2. Confirm that the new disk does not have a partition table:

       # parted /dev/disk/by-hba-slot/sn -s print

   You should see a message about a missing partition table.

3. If the `parted` command displays a partition table, then clear it:

       # dd if=/dev/zero of=/dev/disk/by-hba-slot/sn bs=1M count=100

   Tip:

   You can use this command to restart an operating system disk configuration, if you make a mistake.

4. Create the partition table:

       # parted /dev/disk/by-hba-slot/sn -s mklabel gpt print

5. List the Cylinder, Head, Sector (CHS) partition information of the surviving disk. Thus, if you are partitioning `/dev/disk/by-hba-slot/s0`, then enter `/dev/disk/by-hba-slot/s1` for `/dev/disk/by-hba-slot/s` `m` in the following command:

       # parted /dev/disk/by-hba-slot/sm -s unit chs print

       Model: LSI MR9261-8i (scsi)
       Disk /dev/sda: 243031,30,6
       Sector size (logical/physical): 512B/512B
       BIOS cylinder,head,sector geometry: 243031,255,63.  Each cylinder is 8225kB.
       Partition Table: gpt

       Number  Start         End           File system  Name     Flags
        1      0,0,34        25,127,7      ext3                  raid
        2      25,127,8      21697,116,20  ext3                  raid
        3      21697,116,21  23227,61,35   linux-swap
        4      23227,61,36   243031,29,36  ext3         primary

6. Create partitions 1 to 3 on the new drive by duplicating the partitions of the surviving disk. Issue three commands in this format:

       # parted /dev/disk/by-hba-slot/sn -s mkpart file_system start end

   Use the start and end addresses that you obtained in Step 5 instead of the addresses shown in the following example:

       # parted /dev/disk/by-hba-slot/sn -s mkpart ext3 0,0,34 25,127,7
       # parted /dev/disk/by-hba-slot/sn -s mkpart ext3 25,127,8 21697,116,20
       # parted /dev/disk/by-hba-slot/sn -s mkpart linux-swap 21697,116,21 23227,61,35

7. Create primary partition 4 using the start address obtained in Step 5 and an end address of 100%:

       # parted /dev/disk/by-hba-slot/sn -s mkpart primary ext3 23227,61,36 100%

   Partition 4 stores HDFS data, and this syntax makes the partition as large as possible.

8. Set the RAID flags:

       # parted -s /dev/disk/by-hba-slot/sn set 1 raid
       # parted -s /dev/disk/by-hba-slot/sn set 2 raid

9. Clear the names, using `parted` in interactive mode:

       # parted /dev/disk/by-hba-slot/sn

       GNU Parted 1.8.1
       Using /dev/sdn
       Welcome to GNU Parted! Type 'help' to view a list of commands.
       (parted) name 1 " "
       (parted) name 2 " "
       (parted) name 3 " "
       (parted) quit

10. Complete the steps in "Repairing the RAID Arrays".

### 12.8.2 Repairing the RAID Arrays

After partitioning the disks, you can repair the two logical RAID arrays:

- `/dev/md0` contains `/dev/disk/by-hba-slot/s0p1` and `/dev/disk/by-hba-slot/s1p1`. It is mounted as `/boot`.

- `/dev/md2` contains `/dev/disk/by-hba-slot/s0p2` and `/dev/disk/by-hba-slot/s1p2`. It is mounted as `/` (root).

Caution:

Do not dismount the `/dev/md` devices, because that action shuts down the system.

To repair the RAID arrays:

1. Remove the partitions from the RAID arrays:

       # mdadm /dev/md0 -r detached
       # mdadm /dev/md2 -r detached

2. Verify that the RAID arrays are degraded:

       # mdadm -Q –-detail /dev/md0
       # mdadm -Q –-detail /dev/md2

3. Verify that the degraded file for each array is set to 1:

       # cat /sys/block/md0/md/degraded
       1
       # cat /sys/block/md2/md/degraded
       1

4. Restore the partitions to the RAID arrays:

       # mdadm –-add /dev/md0 /dev/disk/by-hba-slot/snp1
       # mdadm –-add /dev/md2 /dev/disk/by-hba-slot/snp2

5. Check that resynchronization is started, so that `/dev/md2` is in a state of recovery and not idle:

       # cat /sys/block/md2/md/sync_action
       repair

6. To verify that resynchronization is proceeding, you can monitor the `mdstat` file. A counter identifies the percentage complete.

       # cat /proc/mdstat

       Personalities : [raid1]
       md0 : active raid1 sdb1[1] sda1[0]
             204736 blocks [2/2] [UU]

       md2 : active raid1 sdb2[2] sda2[0]
             174079936 blocks [2/1] [U_]
             [============>........]  recovery = 61.6% (107273216/174079936) finish=18.4min speed=60200K/sec

   The following output shows that synchronization is complete:

       Personalities : [raid1]
       md0 : active raid1 sdb1[1] sda1[0]
             204736 blocks [2/2] [UU]

       md2 : active raid1 sdb2[1] sda2[0]
             174079936 blocks [2/2] [UU]

       unused devices: <none>

7. Display the content of /etc/mdadm.conf:

       # cat /etc/mdadm.conf

       # mdadm.conf written out by anaconda
       DEVICE partitions
       MAILADDR root
       ARRAY /dev/md0 level=raid1 num-devices=2 UUID=df1bd885:c1f0f9c2:25d6...
       ARRAY /dev/md2 level=raid1 num-devices=2 UUID=6c949a1a:1d45b778:a6da...

8. Compare the output of the following command with the content of /etc/mdadm.conf from Step 7:

       # mdadm --examine --brief --scan --config=partitions

9. If the UUIDs in the file are different from UUIDs in the output of the `mdadm` command:

   1. Open /etc/mdadm.conf in a text editor.

   2. Select from ARRAY to the end of the file, and delete the selected lines.

   3. Copy the output of the command into the file where you deleted the old lines.

   4. Save the modified file and exit.

10. Complete the steps in "Formatting the HDFS Partition of an Operating System Disk".

### 12.8.3 Formatting the HDFS Partition of an Operating System Disk

Partition 4 (sda4) on an operating system disk is used for HDFS. After you format the partition and set the correct label, HDFS rebalances the job load to use the partition if the disk space is needed.

To format the HDFS partition:

1. Format the HDFS partition as an ext4 file system:

       # mkfs -t ext4 /dev/disk/by-hba-slot/snp4

   Note:

   If this command fails because the device is mounted, then dismount the drive now and skip step 3. See "Prerequisites for Replacing a Failing Disk" for dismounting instructions.

2. Verify that the partition label (such as `/u01` for `s0p4`) is missing:

       # ls -l /dev/disk/by-label

3. Dismount the appropriate HDFS partition, either `/u01` for `/dev/sda`, or `/u02` for `/dev/sdb`:

       # umount /u0n

4. Reset the partition label:

       # tune2fs -c -1 -i 0 -m 0.2 -L /u0n /dev/disk/by-hba-slot/snp4

5. Mount the HDFS partition:

       # mount /u0n

6. Complete the steps in "Restoring the Swap Partition".

### 12.8.4 Restoring the Swap Partition

After formatting the HDFS partition, you can restore the swap partition.

To restore the swap partition:

1. Set the swap label:

       # mkswap -L SWAP-sdn3 /dev/disk/by-hba-slot/snp3
       Setting up swapspace version 1, size = 12582907 kB
       LABEL=SWAP-sdn3, no uuid

2. Verify that the swap partition is restored:

       # bdaswapon; bdaswapoff
       Filename                          Type            Size    Used    Priority
       /dev/sda3                         partition       12287992        0       1
       /dev/sdb3                         partition       12287992        0       1

3. Verify that the replaced disk is recognized by the operating system:

       $ ls -l /dev/disk/by-label
       total 0
       lrwxrwxrwx 1 root root 10 Aug  3 01:22 BDAUSB -> ../../sdn1
       lrwxrwxrwx 1 root root 10 Aug  3 01:22 BDAUSBBOOT -> ../../sdm1
       lrwxrwxrwx 1 root root 10 Aug  3 01:22 SWAP-sda3 -> ../../sda3
       lrwxrwxrwx 1 root root 10 Aug  3 01:22 SWAP-sdb3 -> ../../sdb3
       lrwxrwxrwx 1 root root 10 Aug  3 01:22 u01 -> ../../sda4
       lrwxrwxrwx 1 root root 10 Aug  3 01:22 u02 -> ../../sdb4
       lrwxrwxrwx 1 root root 10 Aug  3 01:22 u03 -> ../../sdc1
       lrwxrwxrwx 1 root root 10 Aug  3 01:22 u04 -> ../../sdd1
            .
            .
            .

4. If the output does not list the replaced disk:

   - On Linux 5, run `udevtrigger`.

   - On Linux 6, run `udevadm trigger`.

   Then repeat step 3. The `lsscsi` command should also report the correct order of the disks.

5. Complete the steps in "Restoring the GRUB Master Boot Records and HBA Boot Order".

### 12.8.5 Restoring the GRUB Master Boot Records and HBA Boot Order

After restoring the swap partition, you can restore the Grand Unified Bootloader (GRUB) master boot record.

The `device.map` file maps the BIOS drives to operating system devices. The following is an example of a device map file:

    # this device map was generated by anaconda
    (hd0)     /dev/sda
    (hd1)     /dev/sdb

However, the GRUB device map does not support symbolic links, and the mappings in the device map might not correspond to those used by `/dev/disk/by-hba-slot`. The following procedure explains how you can correct the device map if necessary.

To restore the GRUB boot record:

1. Check which kernel device the drive is using in slot1

       # ls -ld /dev/disk/by-hba-slot/s1
       lrwxrwxrwx 1 root root 9 Apr 22 12:54 /dev/disk/by-hba-slot/s1 -> ../../sdb

2. If the output displays`/dev/sdb` as shown in step 1, then proceed to the next step (open GRUB).

   If another device is displayed, such as `/dev/sdn`, then you must first set `hd1` to point to the correct device:

   1. Make a copy of the `device.map` file:

          # cd /boot/grub
          # cp device.map mydevice.map
          # ls -l *device*
          -rw-r--r-- 1 root root 85 Apr 22 14:50 device.map
          -rw-r--r-- 1 root root 85 Apr 24 09:24 mydevice.map

   2. Edit `mydevice.map` to point `hd1` to the new device. In this example, `s1` pointed to `/deb/sdn` in step 1.

          # more /boot/grub/mydevice.map
          # this device map was generated by bda install
          (hd0)     /dev/sda
          (hd1)     /dev/sdn

   3. Use the edited device map (`mydevice.map`) in the remaining steps.

3. Open GRUB, using either `device.map` as shown, or the edited `mydevice.map`:

       # grub --device-map=/boot/grub/device.map

           GNU GRUB  version 0.97  (640K lower / 3072K upper memory)

        [ Minimal BASH-like line editing is supported.  For the first word, TAB
          lists possible command completions.  Anywhere else TAB lists the possible
          completions of a device/filename.
       ]

4. Set the root device, entering hd0 for `/dev/sda`, or hd1 for `/dev/sdb`:

       grub> root (hdn,0)

       root (hdn,0)
        Filesystem type is ext2fs, partition type 0x83

5. Install GRUB, entering hd0 for `/dev/sda`, or hd1 for `/dev/sdb`:

       grub> setup (hdn)

       setup (hdn)
        Checking if "/boot/grub/stage1" exists... no
        Checking if "/grub/stage1" exists... yes
        Checking if "/grub/stage2" exists... yes
        Checking if "/grub/e2fs_stage1_5" exists... yes
        Running "embed /grub/e2fs_stage1_5 (hdn)"... failed (this is not fatal)
        Running "embed /grub/e2fs_stage1_5 (hdn,0)"... failed (this is not fatal)
        Running "install /grub/stage1 (hdn) /grub/stage2 p /grub/grub.conf "... succeeded
       Done.

6. Close the GRUB command-line interface:

       grub> quit

7. Ensure that logical drive `L0` (L + zero) is set as the boot drive in the HBA:

       # MegaCli64 -AdpBootDrive -get a0
       Adapter 0: Boot Virtual Drive - #0 (target id - 0).

8. If the previous command does not report `L0` or virtual drive 0 target 0, then enter:

       # MegaCli64 AdpBootDrive set L0 a0

9. Ensure that the auto-select boot drive feature is enabled:

       # MegaCli64 adpBIOS EnblAutoSelectBootLd a0
       Auto select Boot is already Enabled on Adapter 0.

10. Check the configuration. See "Verifying the Disk Configuration."

## 12.9 Configuring an HDFS or Oracle NoSQL Database Disk

Complete the following instructions for any disk that is not used by the operating system. See "Identifying the Function of a Disk Drive."

To configure a disk, you must partition and format it.

Note:

Replace snp1 in the following commands with the appropriate symbolic name, such as s4p1.

To format a disk for use by HDFS or Oracle NoSQL Database:

1. Complete the steps in "Replacing a Disk Drive," if you have not done so already.

2. Partition the drive:

       # parted /dev/disk/by-hba-slot/sn -s mklabel gpt mkpart primary ext4 0% 100%

3. Format the partition for an ext4 file system:

       # mkfs -t ext4 /dev/disk/by-hba-slot/snp1

4. Reset the appropriate partition label to the missing device. See Table 12-2.

       # tune2fs -c -1 -i 0 -m 0.2 -L /unn /dev/disk/by-hba-slot/snp1

   For example, this command resets the label for `/dev/disk/by-hba-slot/s2p1` to `/u03`:

       # tune2fs -c -1 -i 0 -m 0.2 -L /u03 /dev/disk/by-hba-slot/s2p1
       Setting maximal mount count to -1
       Setting interval between checks to 0 seconds
       Setting reserved blocks percentage to 0.2% (976073 blocks)

5. Verify that the replaced disk is recognized by the operating system:

       $ ls -l /dev/disk/by-label
       total 0
       lrwxrwxrwx 1 root root 10 Aug  3 01:22 BDAUSB -> ../../sdn1
       lrwxrwxrwx 1 root root 10 Aug  3 01:22 BDAUSBBOOT -> ../../sdm1
       lrwxrwxrwx 1 root root 10 Aug  3 01:22 SWAP-sda3 -> ../../sda3
       lrwxrwxrwx 1 root root 10 Aug  3 01:22 SWAP-sdb3 -> ../../sdb3
       lrwxrwxrwx 1 root root 10 Aug  3 01:22 u01 -> ../../sda4
       lrwxrwxrwx 1 root root 10 Aug  3 01:22 u02 -> ../../sdb4
       lrwxrwxrwx 1 root root 10 Aug  3 01:22 u03 -> ../../sdc1
       lrwxrwxrwx 1 root root 10 Aug  3 01:22 u04 -> ../../sdd1
            .
            .
            .

6. If the output does not list the replaced disk:

   - On Linux 5, run `udevtrigger`.

   - On Linux 6, run `udevadm trigger`.

   Then repeat step 5. The `lsscsi` command should also report the correct order of the disks.

7. Mount the HDFS partition, entering the appropriate mount point:

       # mount /unn

   For example, `mount /u03`.

8. If you are configuring multiple drives, then repeat the previous steps.

9. If you previously removed a mount point in Cloudera Manager for an HDFS drive, then restore it to the list.

   1. Open a browser window to Cloudera Manager. For example:

      `http://bda1node03.example.com:7180`

   2. Open Cloudera Manager and log in as `admin`.

   3. On the Services page, click hdfs

   4. Click the Instances subtab.

   5. In the Host column, locate the server with the replaced disk. Then click the service in the Name column, such as datanode, to open its page.

   6. Click the Configuration subtab.

   7. If the mount point is missing from the Directory field, then add it to the list.

   8. Click Save Changes.

   9. From the Actions list, choose Restart.

10. Check the configuration. See "Verifying the Disk Configuration."

## 12.10 Verifying the Disk Configuration

Before you can reinstall the Oracle Big Data Appliance software on the server, you must verify that the configuration is correct on the new disk drive.

To verify the disk configuration:

1. Check the software configuration:

       # bdachecksw

2. If there are errors, then redo the configuration steps as necessary to correct the problem.

3. Check the `/root` directory for a file named `BDA_REBOOT_SUCCEEDED`.

4. If you find a file named `BDA_REBOOT_FAILED`, then read the file to identify and fix any additional problems.

5. Use this script to generate a `BDA_REBOOT_SUCCEEDED` file:

       # /opt/oracle/bda/lib/bdastartup.sh

6. Verify that `BDA_REBOOT_SUCCEEDED` exists. If you still find a `BDA_REBOOT_FAILED` file, then redo the previous steps.